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1  Introduction 


Our  approach  is  to  provide  an  environment  which  captures  the  student's  interest  by  providing 
an  engaging  story  and  provides  educational  value  by  mapping  curriculum  elements  in  to  the  fabric 
of  that  story.  Both  the  story  and  the  curriculum  are  realized  in  the  creation  of  a  game  in  a  virtual 
world  with  which  the  student  can  interact;  this  overarching  concept  is  illustrated  in  Figure  2-1.  In 
this  way  we  provide  the  student  a  context  in  an  engaging  virtual  world  that  will  require 
understanding  curriculum  objectives  in  order  succeed. 

1.1  Project  Objectives  and  Requirements 

The  PhysicsFun4k24,  or  simply  PhysicsFun,  project  has  defined  the  following  objectives: 

•  Formulate  a  gaming  environment  for  learning  a  basic  set  of  physics  principles  where 
students  are  fully  "immersed"  in  lessons,  enabled  to  discovery  principles  and  allowed 
to  identify  limits  and  extension  of  principles  through  experimentation; 

•  Create  a  prototype  of  that  environment  for  students  to  experientially  explore  and 
understand  a  basic  set  of  physics  principles,  where  students  interact  with  the  virtual 
world  in  a  multi-sensory  manner,  and 

•  Provide  assessments  embedded  into  the  gaming  environment  that  examines  students' 
attitudes,  confidence,  and  knowledge  of  physics  and  the  effectiveness  of  the  immersive 
gaming  environment. 

Beyond  the  satisfaction  of  these  objectives,  there  are  also  certain  requirements  that  must  be 
met  in  order  for  the  concepts  developed  to  be  successfully  deployed  and  used  in  early  elementary 
STEM  education: 

•  Present  an  affordable  system  that  employs  state-of-the-art  3D  input  devices  and 
immersive  gaming  technology  and  leverages  recent  advances  in  computing  systems, 
visual  displays,  and  computer-generated  imagery  software. 

•  Create  software  environment  designed  to  operate  on  a  range  of  hardware 
configurations  including  laptop,  PC  or  tabletop  workstations  and  positioned  to  evolve 
with  changing  technology. 

•  Coordinate  the  PhysicsFun  curriculum  with  teacher,  school  system,  state  and  national 
content  and  standards,  and  allow  that  curriculum  to  be  easily  updated  and  expanded. 

1.2  Deliverables 

This  phase  of  the  PhysicsFun  project  only  lays  the  ground  work  for  meeting  these  objectives  and 
provides  foundational  elements  to  research  and  build  complete  systems  in  future  work.  Adaptive 
Methods  will  deliver  the  following  products  as  the  results  of  these  efforts: 
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•  Research  results  that  document  the  Immersive  STEM  Methodology1  a  systematic  way  to 
map  curriculum  to  immersive  game  elements  and  to  controls  game  progress  as  students 
learn  the  curriculum. 

•  A  seedling  game2,  including  the  software  and  supporting  hardware  suite  with  multiple 
inputs  concepts  and  visualization  capabilities,  which  can  be  deployed  and  played  as  a  proof- 
of-principle. 


2  Immersive  STEM  Methodology 

Science  and  technology 
motivation  and  education  is  the 
primary  goal  of  PhysicsFun.  This 
part  of  the  research  focuses  on 
the  methods  and  techniques  for 
effectively  mapping  a  STEM 
curriculum  into  an  immersive 
environment  to  accomplish  this 
end. 

This  methodology  is 
architected  around  three 
principle  concepts,  show  in 
Figure  2-1: 

•  Curriculum  Space  -  The 

curriculum  space 
which  provides 
grounding  for  the 

information  being  learned.  It  identifies  the  individual  objectives,  prerequisites,  optimal 
and  alternate  pathways  for  learning. 

•  Story  Space  -  The  story  space  provides  situation  and  context  through  which  curriculum  is 
experienced  and  explored.  It  is  defined  by  a  series  of  events,  locations,  characters  in  a 
narrative  context. 

•  Virtual  World  -  The  story  is  realized  using  a  gaming  engine  with  appropriate  mechanics, 
simulating  the  virtual  world  as  the  student  interacts  with  it.  Specific  information  relating 
to  the  curriculum  is  presented  to  the  student  or  available  for  the  student  to  investigate. 


Figure  2-1  -  Adaptive  Methods’  Immersive  STEM  concept,  the 
student  must  apply  current  knowledge  of  a  defined  curriculum  to 
succeed  in  the  virtual  world;  knowledge  that  the  student  lacks  can 
be  gained  while  participating  in  the  associated  story  presented  by 
the  virtual  world. 


1  CLIN  A0002 

2  CLIN  A0003,  CLIN  A0004,  and  CLIN  A0005 
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The  students  experience  and  interact  as  their  avatars  move  through  the  environment, 
seeing,  hearing,  finding  information  base  on  where  they  are  and  what  they  do. 

The  process  starts  with  a  concrete  curriculum  plan,  a  standard  of  learning  or  a  classroom  lesson 
plan.  Typically,  the  plan  has  a  hierarchical  outline,  collecting  related  topics  and  concepts  into 
logical  groupings.  The  hierarchy  of  topics  becomes  definitions  for  a  hierarchy  of  modules,  sub- 
modules  and  lessons. 

The  modules  and  lessons  are  defined  by: 

•  Prerequisites  -conditions  that  must  be  meet  before  this  module  maybe  started, 

•  Entry  assessment  -  a  pre-assessment  of  the  student's  understanding  of  the  modules 
content, 

•  Content  -  the  facts,  concepts,  skills  and  ideas  contained  in  this  module,  and 

•  Exit  assessment  -  a  post-assessment  of  the  students  understanding  of  the  modules 
content. 


educational  content 

Figure  2-2  -  The  concept  of  mapping  curriculum  to  game  elements  and  story. 


The  modules  are  organized  into  the  curriculum  graph  based  upon  the  prerequisites  of  each 
module.  Figures  2-2  shows  a  notional  curriculum  with  six  modules.  Various  types  of  prerequisites 
within  the  curriculum  graph  are  possible.  For  example,  module  M4  has  an  "and"  constraint 
requiring  modules  M2  and  M3  both  be  completed  before  starting  M4;  M6  has  an  "or"  constraint 
needing  either  M4  or  M5  to  begin.  Sub-modules,  such  as  M3  A  and  B,  are  alternate  presentations 
of  the  same  content.  The  path  that  the  student  takes  through  the  curriculum  graph  is  flexible,  as 
illustrated  by  the  pink  and  green  paths  through  the  modules  in  Figure  2-2. 

The  path  that  a  student  takes  through  the  curriculum  graph  is  flexible.  Lessons  are  similar  to 
modules  and  represent  the  lowest  level  of  detail  in  the  curriculum.  Notional  Lessons  are  shown  in 
Figure  2-3,  in  a  graph  from  completely  analogous  to  one  for  modules.  In  this  case,  the  entry  and 
exit  assessments  are  shown  for  the  containing  module. 


3 
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The  curriculum  graph,  with  its 
modules  and  lessons,  is  the 
framework  around  which  the 
story  is  woven  and  the 
inspiration  for  game  elements. 
Each  lesson,  and  some  lower 
level  modules,  will  be  mapped  to 
elements  in  the  story  and  one  or 
more  levels  in  the  game. 


3  The  Prototype  Game 

This  section  describes  the  concept  and  high  level  design,  of  the  prototype's  game  "Jump'n 
Tadpoles". 

3.1  Story  Space-  Navy  Leap  Frogs 

Jump'n  Tadpoles  presents  the  opportunity  for  students  to  learn  what  it  takes  to  be  a  member  of 
the  U.S.  Parachute  Team  "Navy  Leap  Frogs"3,  and  gain  an  understanding  of  physics  along  the  way. 
During  the  game  students  will  experience  the  effects  of  gravity  and  aerodynamic  drag  on  a  series  of 
virtual  skydiving  training  missions.  They  will  learn  what  terminal  velocity  means,  how  to  maximize 
their  velocity  and  control  maneuvers  by  understanding  the  relationships  between  air  speed  and 
their  own  body  positions. 

3.2  Curriculum  Space  -  Force  and  Motion 

We  have  developed  a  single  physics  curriculum  module  based  on  content  collected  from  The 
State  of  Virginia  Science  Standards  of  Learning. 

3.2.1  First  Grade,  parts  1.2  a,  c,  and  d 


L31 


Lessons  within  module  are 
the  minimum  “atomic" 
component  of  curriculum. 


ippnrtunity  to  assess 
student  knowledge 
before,  after  or  during 
Module's  Lessons. 


Figure  2-3  -  The  concept  of  mapping  curriculum  to  game  elements 

and  story. 


Force,  Motion,  and  Energy 

1.2  The  student  will  investigate  and  understand  that  moving  objects  exhibit  different  kmds  of  motion. 
Key  concepts  include 

a)  objects  may  have  straight,  circular,  and  back-and-fortli  motions; 

b)  objects  may  vibrate  and  produce  sound; 

c)  pushes  or  pulls  can  change  the  movement  of  an  object;  and 

d)  the  motion  of  objects  may  be  observed  m  toys  and  m  playground  activities. 


3  http://www.leapfrogs.navy.mil/ 
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3.2.2  Fourth  Grade,  Parts  4.2  a,  b  and  c 


Force,  Motion,  and  Energy 

4.2  The  student  will  investigate  and  understand  characteristics  and  mteraction  of  moving  objects.  Key 
concepts  include 

a)  motion  is  described  by  an  object’s  direction  and  speed; 

b)  forces  cause  changes  m  motion; 

c)  friction  is  a  force  that  opposes  motion;  and 

d)  moving  objects  have  kinetic  energy. 


The  module  will  introduce  students  to  Newton's  first  and  second  laws  of  motion. 

•  First  Law  -  Motion  only  changes  if  forces  pushing  and  pulling  are  out  of  balance. 

•  Second  Law  -  A  motions  rate  of  change  is  based  on  the  size  of  the  push  or  pull. 

Forces  are  introduced  using  gravity,  things  fall  toward  the  earth,  and  drag,  things  moving 
through  the  air  produce  friction  which  resists  motion.  Velocity  and  acceleration  are  introduced  as 
components  of  motion.  Velocity  is  the  basis  for  motion,  when  something  has  a  velocity  it  is  moving 
in  a  certain  direction  with  a  certain  speed.  Acceleration  is  the  change  in  velocity  over  time. 
Velocities  either  increase  (go  faster)  or  decrease  (slow  down)  or  change  direction  (same  speed). 

A  summary  of  physics  concepts  and  values  used  during  game  play,  underlying  what  is  visually 
presented  to  the  students,  is  summarized  in  the  table  below. 


Parameter 

Value 

Notes 

Time 

Time  during  jump,  in 
seconds 

Time  reset  to  0.0  when  a  jump  begins. 

Altitude 

Varies  during  jump 

Altitude  reset  to  constant  (~1000m) 
when  jump  begins;  integration  of 

Velocity  over  Time. 

Velocity 

Varies  during  jump 

Integration  of  Acceleration  over  Time 

Acceleration 

Varies  during  jump 

Force  divided  by  Mass 

Mass 

30kg 

The  "weight"  of  the  avatar.  (66  lbs),  held 
as  a  constant. 

Weight 

294  kg-m/sA2 

The  force  of  Gravity  acting  on  Mass 

Force 

Drag  minus  Weight 

The  net  forces  acting  on  the  avatar 
(positive  is  up) 

Gravity 

9.8  m/sA2 

Gravitational  Constant  on  Earth 
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Drag 

Coefficient  *  VelocityA2 

The  force  due  to  moving  through  the  air. 

Coefficient 

0.5*Aero*Density*Area 

Values  from~  0.12  (arms  out)  to  ~0.048 
(arms  in)  for  nominal  conditions. 

Aero 

0.039  kg-m 

Aerodynamic  Constant 

Density 

1.23  kg/mA3 

Density  Earth's  atmosphere  (ignoring 
altitude). 

Area 

Varies  from  5.0  to  2.0  mA2 

Based  on  player  position  -  arms  in  or  out 
/  up  or  down  -  determines  Area  which 
provides  a  contribution  to  Coefficient. 

3.3  Game  Space  -  Virtual  World 

Students'  experiences  from  commercial  off-the-shelf  (COTS)  games  and  with  web  content  have 
influenced  their  expectations  of  quality  in  immersive  interfaces.  In  order  for  these  expectations  to 
be  met,  high  quality  design  and  planning  must  be  incorporated  into  the  virtual  environment.  This  is 
accomplished  in  the  production  and  review  of  storyboards,  concept  graphics  and  game  play 
aspects,  examples  are  included  below. 

The  game  is  designed  to  maximize  its  immersive  nature.  For  example,  menu  selection  is 
minimized,  using  motion  and  gestures  were  possible;  displays  are  a  part  of  the  environment,  heads 
up  displays  rather  that  separate  delineated  ones  are  preferred.  Maximum  use  is  made  of  the  visual 
3D  and  audio  capabilities. 

3.4  Putting  it  all  together  -  Jump'n  Tadpoles 

The  above  three  elements,  story,  curriculum  and  virtual  world,  will  be  combined  to  produce  the 
"Jump'n  Tadpoles"  game.  The  game  will  consist  of  one  curriculum  module,  with  four  lessons,  as 


Story  -  Leap  Frog  Training 


Figure  3-1  -  The  story  provides  a  simple  linear  progression  of  game  level  mapped  to  a  single 

physiscs  module  with  four  lessons. 
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shown  in  Figure  3-1,  each  lesson  is  mapped  to  a  single  level  in  the  game.  This  is  a  very  simple 
curriculum-to-game  mapping  that  shows  proof-of-concept.  The  students  interact  with  and  progress 
through  the  game  using  gesture  and  motion  based  APIs  with  both  worn  gloves  and  position 
tracking  component  as  options.  They  are  presented  with  a  3D  presentation  of  the  virtual  world. 

The  game  is  organized  internally  as  a  series  of  different  scenes,  shown  conceptually  in  Figure  3- 


2.  These  scenes  in  the  game  represent  one  or  more  states  or  modes  in  the  game  play.  The  green 
scenes  are  primarily  administrative,  the  blue  are  help  and  practice,  while  the  yellow  are  actual 
game  play.  Note  that  the  game  play  scenes  are  used  slightly  differently  for  each  of  levels  one,  two 
and  three. 

The  arcs  represent  events,  and  the  associated  transitions  between  scenes  and  states.  They  may 
be  generated  either  internally  in  game,  such  a  "land"  when  the  student  avatar  reaches  the  ground, 
or  produced  externally  by  gestures,  such  as  "help",  "jump"  or  "pull". 

In  the  following  sections,  we  provide  concept  artwork,  Figures  3-3  to  3-6,  and  discuss  each  level 
of  play  in  the  game.  This  includes  the  elements  from  Story,  Curriculum  and  Game  that  are  at  play  in 
each  level. 
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U.S.  Navy  Skydiving  Recruits 


3235?  )  S> 


(b) 

Figure  3-3  Concept  art  for  the  icon  styles  (a)  and  student  avatar  selection  (b). 
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3.4.1  Level  0-  Briefing 

This  is  the  introductory  briefing  level  that  provides  an  introduction  to  the  game,  including  the 
overall  mission,  objectives  for  each  level,  vocabulary  and  game  controls  (including  "Practice" 
controls) 
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Figure  3-4- 


Storyboard  concept  for  the  Briefing  Room  scene  illustrating  the  explaination  of  Terminal 

Velocity  as  an  example. 


Story  Elements 

You  are  a  US  Navy  sailor,  competing  to  be  a  Navy  Leap  Frog.  You  are  learning  the  science  and 
skills  required  to  effectively  and  safely  skydive. 

Curriculum  Elements 

•  Vocabulary: 

o  Area 

o  Forces,  Drag,  Gravity 
o  Acceleration,  Velocity 
o  Terminal  Velocity 
o  Atmosphere 

Prerequisites 

•  None. 
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Game  Elements 

•  Controls 

o  Wave,  Select,  Choose 
o  Jump,  Spread  eagle,  Dive,  Turn  and  Aim 

Assessment 

•  Skills  to  use  gesture  interface  to  control  game. 

3.4.2  Level  1  -  Free  Falling 
Story  Elements 

You  need  to  be  able  to  move  fast.  You  need  to  determine  how  to  reach  your  greatest  speed 
during  your  decent.  You  can  get  faster  by  getting  smaller. 

Curriculum  Elements 

The  forces  of  air  resistance  and 
gravity,  free  fall,  and  terminal 
velocity 

•  Drag  and  gravity 

•  Balancing  forces  at 
terminal  velocity 

•  Affect  of  position  on 
drag 

Prerequisites 

•  Level  0  -  Successful  use 
of  controls  to  perform 
practice  actions. 

Game  Elements 

•  Beginning 

o  Player  ready  to 
jump  from  air 
plane  at  specific 
altitude  selected 
via  player  input, 
default  to  be 
determined  by 
game  testing 

•  Controls  -  Input  via  body  position 

o  Jump  starts 

o  Spread  eagle  -  avatar  to  flat  position 
o  Straight,  arms  aside  -  avatar  to  dive 
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o  Leg(s)  or  arm(s)  outstretched  -  rotational  physics  determines  path 

•  Feedback 

o  Arrows  represent  forces  due  to  air  resistance  induce  drag  and  gravity 
o  When  both  arrows  are  the  same  size,  terminal  velocity  achieved  until  the  player 
changes  position,  arrows  flash,  and  top  speeds  are  recorded, 
o  Speedometer  shows  current  speed,  arrow  show  up-getting  faster,  down-getting 
slower,  dash  -  constant. 

Assessment 

•  Understand  the  position  to  minimize  the  drag  from  wind  resistance  needed  to  maximize 
the  player's  terminal  velocity. 

3.4.3  Level  2  -Landing  Safe 

Introduces  player  to  the  forces  of  air  resistance  depending  on  overall  shape  (area)  and  how  a 
parachute  lowers  terminal  velocity. 

Story  Elements 

You  need  to  get  down  fast,  but  you  must  land  safely.  You  can  only  get  so  slow  by  making 
yourself  big,  you  need  a  parachute  to  go  even  slower. 

Curriculum  Elements 

Opening  parachute  changes  drag  (makes  a  much  larger)  slowing  you  down  -  same  weight  but 
bigger  area;  body  position  has  less  or  no  effect  with  parachute  deployed. 

•  Drag  verse  Area 

•  Shape  does  not  affect  gravity,  only  mass 

Prerequisites 

Level  1  -  Reach  a  specified  terminal  velocity. 

Game  Elements 

•  Beginning 

o  Player  ready  to  jump  from  air  plane  at  specific  altitude  selected  via  player  input, 
default  to  be  determined  by  game  testing 

•  Controls  -  Input  via  body  position 

o  Jump  starts 

o  Spread  eagle  -  avatar  to  flat  position 
o  Straight,  arms  aside  -  avatar  to  dive 

o  Leg(s)  or  arm(s)  outstretched  -  rotational  physics  determines  path 
o  Specific  gesture  (e.g.  Pull  hands  over  head  and  back  to  chest)  to  deploy  chute 

•  Feedback 

o  Time  counts  down  to  appropriate  altitude 

o  If  pulled  in  time,  float  down  with  chute  -  thumbs  up 

o  If  not  pulled  in  time,  Wiley  E  Coyote-like  ending  -  offer  to  replay 
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Assessment 

•  Understand  that  larger  areas  produce  larger  drag  forces, 
o  Use  parachute  to  slow  down  and  land. 

3.4.4  Level  3  -  Hitting  the  Mark 
Story  Elements 

Leap  Frogs  must  be  able  to  land  where  they  want  to  land,  so  they  need  to  know  how  to  hit  the 
mark.  Changing  your  shape  can  change  which  way  air  is  pushing  you,  so  you  can  steer  with  you 
arms. 


Copyright©2012  Immersive  3d,  LLC 


Figure  3-6  -  Storyboard  concept  for  the  Hitting  the  Mark;  shown  as  part  of  the  opening 
splash  screen.  Foreshadows  a  goal  for  the  students  to  move  towards. 


Curriculum  Elements 

•  Forces  have  a  direction. 

Prerequisites 

•  Level  2  -  Making  a  safe  landing. 
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Game  Elements 

•  Beginning 

o  Player  ready  to  jump  from  air  plane  at  specific  altitude  selected  via  player  input, 
default  to  be  determined  by  game  testing 

•  Controls 

o  Same  as  Free  Fall 

•  Feedback 

o  Score  display 

o  Distance  to  safe  chute  deployment  altitude 

o  If  pulled  in  time,  float  down  with  chute  -  thumbs  up  -  score  saved,  if  top  score 
o  If  not  pulled  in  time,  Wiley  E  Coyote-like  ending  -  offer  to  replay  -  no  score  saved 

Assessment 

•  Understand  that  drag  forces  have  direction  in  order  to  steer. 

o  Collect  specified  number  of  points  and  land  safely. 

4  Measurement  and  Assessment 

Assessment  is  provided  at  each  level  through  scoring.  In  PhysicsFun  the  scoring  system  is 
designed  around  enumerated  learning  objectives.  PhysicsFun  implements  an  in-process  game 
statistics  engine  which  measures  and  records  information  on  a  per  Player  basis  for  assessment  and 
review.  The  recorded  data  will  be  used  to  provide  assessment  of  PhysicsFun's  defined  research 
objectives. 

According  to  Interaction  Designer  John  Ferguson,  figuring  out  if  a  Human  actual  learned 
something  is  a  difficult  task.  Assessing  player  learning  in  game  play  presents  three  major 
challenges  (Ferguson,  2012): 

1.  How  do  you  show  that  Players  are  learning  what  you  claim  they  are  learning? 

2.  How  do  you  know  that  what  you  are  measuring  is  what  you  think  you  are  measuring? 

3.  Games  that  act  too  much  like  a  classroom,  with  quizzes  interrupt  a  player's  experience. 

In  order  for  the  PhysicsFun  to  be  a  successful  tool  that  augments  classroom  learning  it  needs  to 
incorporate  learning  assessment,  and  evaluation  capabilities  into  Game  Play. 

4.1  Physics  Fun 

Assessment  and  evaluation  in  Jump'n  Tadpoles,  (and  Future  PhysicsFun  Game  installments),  is 
based  upon  proven  Game  Play  pedagogical  and  psychometric  measurement  strategies. 

The  focus  is  to  create  an  environment  where  learning  is  definable,  quantifiable  and  measurable. 
PhysicsFun's  pedagogical  approach  incorporates  science  and  technology  lesson  plans  and 
assessments  seamlessly  into  the  game  play  sequence. 
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By  employing  an  integrated  approach  PhysicsFun  is  not  bound  to  traditional  classroom 
evaluation  applications  (e.g.  the  administration  of  series  simple  multiple-choice  questions  at  the 
conclusion  of  an  interactive  Game  Play  scenario). 

In  PhysicsFun,  dynamic  Game  Play  measurements  are  employed  to  support  Game  Play 
evaluation/assessment,  to  control  player  advancement,  and  to  provide  post  Game  Play  statistical 
analysis. 

4.2  Assessment  Techniques 

Three  types  of  assessment  are  commonly  used  in  educational  games: 

•  Completion  Assessment  -  Did  the  player  complete  the  lesson  or  pass  the  test? 

•  In-Process  Assessment  -  How  did  the  player  choose  his  or  her  actions?  Did  he  or  she 
change  their  mind?  If  so,  at  what  point? 

•  Teacher  Evaluation  -  Based  on  observations  of  the  player,  does  the  teacher  think  the 
player  now  knows/understands  the  material? 

According  to  Kevin  Corti,  Managing  Director  of  PixeLearning,  "Completion  assessment  is  the 
simplest  form  of  assessment:  Did  the  Player  complete  the  game?  This  is  equivalent  to  asking,  "Did 
the  student  get  the  right  answer?"  Since  games  are  simulations,  this  simple  criterion  could  be  the 
first  indicator  that  the  Player  sufficiently  understands  the  subject  taught.  This  is  not  the  same  as 
asking,  "Did  the  student  attend  every  class?"  Because  games  require  interaction  by  the  Player  with 
the  material,  completing  the  game  signifies  more  learning  progress  and  comprehension  than 
passively  attending  a  class  in  a  typical  classroom  setting."  (Corti,  2012) 

It  is  important  to  understand  that  completion  assessment  cannot  distinguish  whether  a  Player 
understands  the  game  lesson,  or  has  just  learned  how  to  beat  the  game. 

In-process  assessment  is  analogous  to  teacher  observations  of  the  student  as  the  student 
performs  the  task  or  takes  the  test.  In  PhysicsFun  our  in-process  assessment  measures: 

•  Time  required  to  complete  a  Game  level; 

•  Mistakes; 

•  Self-corrections  actions. 

Teacher  evaluation  is  a  combination  of  both  completion  assessment  and  in-process  assessment. 
Teacher  evaluation  can  also  include  observation  of  Players  in  action.  Multiplayer  video  games  often 
include  "observer  modes"  that  could  be  used  for  this,  both  by  the  teacher  and  other  students.4 

4.3  Post  Games  Quiz 

In  PhysicsFun  Assessment  starts  pre-game,  runs  all  the  way  through  the  game.  Applying  a 
simple  post-game  multiple-choice  questionnaire  at  the  conclusion  of  a  Players  experience  is  not 
always  the  best  measurement  technique.  Players  come  to  understand  the  connection  between 
their  in-game  actions  and  the  outcomes. 

4  Jim  Brazell,  consulting  analyst  at  the  Digital  Media  Collaboratory  (DMC)  in  the  1C2  Institute  at  the  University  of  Texas  at  Austin, 
www.ic2.utexas.edu. 
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The  application  of  a  simple  assessment  engine  supports  player  assessment  and  post-play 
evaluation.  If  a  player  performs  X  what  was  the  basis  for  the  action,  why  not  Y?  The  assessment 
engine  in  PhysicsFun  records  individual  player  action.  Player  actions  can  be  evaluated  to 
determine  how  well  the  player  understood  the  targeted  learning  objectives. 

4.4  Learning  Assessment 

Science  and  technology  education  is  the  primary  goal  of  PhysicsFun.  All  Leap  Frogs  and  Future 
PhysicsFun  Game  installments  are  architected  around  set  of  enumerated  learning  objectives 
derived  from  age-appropriate  science  and  technology  curriculum. 

Each  installment  in  PhysicsFun  employs  a  multi-level  game  format.  In  level  0,  the  tutorial: 

•  The  player  is  introduced  to  a  enumerate  set  learning  objects; 

•  The  player  is  presented  with  basics  instructions  of  how  to  control  and  interact  with  the 
Game; 

•  The  player  is  provided  and  explanation  of  how  the  series  of  interface  actions  map  to  the 
enumerated  learning  objectives; 

•  The  player  is  provided  with  a  limited  Game  Play  scenario  that  demonstrates  game 
controls;  if  sufficient  mastery  of  the  controls  is  performed  the  Player  can  proceed  to  the 
next  level  of  Play. 

In  Level  0  a  mission  brief  introduces  specific  Game  Play  criteria  and  reinforces  the  learning 
objectives. 

PhysicsFun  employs  pass/fail  mechanisms  to  enforce  Prerequisites  through  subsequent  levels. 
Assessment  is  provided  at  each  level  through  scoring.  The  scoring  system  is  design  around  the 
enumerated  learning  objectives. 

Game  play  in  PhysicsFun  places  less  emphasis  on  rote  memorization  of  facts  than  applying 
traditional  learning  methods.  Open-ended  Game  scenarios  are  presented  to  players  that  wide 
range  of  possible  solutions  all  of  which  could  be  correct.  Measurements  are  included  for  both  in 
game  evaluation  /  control  of  Prerequisites  and  after  play  statistical  analysis.  The  linkages  of  content 
to  game  play  known,  but  cannot  be  directly  observed.  That  is  how  we  know  the  student 
understands  the  concepts  in  the  curriculum  based  on  actions  in  the  game.  To  support  inferential 
assessment  the  mapping  of  content  will  be  mapped  to  in  game  action  allowing  only  students  with 
knowledge  of  concept  X  to  succeed  at  game  element  Z.  Ideally  success  with  game  element  Z 
implies  understanding  of  X. 

4.5  Logging  Measures 

In  order  to  provide  data  for  longer  term,  broader  analysis  of  student  comprehension  and 
engagement  it  is  necessary  to  provide  a  systematic  recording  of  students'  behavior  in  game  into 
persistent  log.  Each  measure  that  is  recorded  must  be  well  designed  and  meet  a  minimum  set  of 
requirements  to  support  rigor  in  subsequent  analysis;  each  measure  shall  be: 
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•  Atomic  -  uninterrupted  events  and  actions  in  the  game. 

•  Clearly  defined  -  unambiguous  &  usable  -  records  a  significant  event 

•  Have  sufficient  Context  -records  all  relevant  contextual  data 

•  Efficient  -  suitable  for  in  game  use  as  well  as  post  play  analysis 

In  addition  to  these  specific  requirements,  the  measures  in  aggregate  must  answer  the 
questions  being  asked  by  the  research.  Our  objectives  are  twofold,  to  motivate  students  interest  in 
science  and  technology  and  to  provide  skill  necessary  to  succeed.  Target  behaviors  that  reflect  skills 
needed  to  be  successful  may  include:  problem  solving,  situation  awareness,  decision  making, 
conceptual  learning  and  procedural  learning. 

PhysicsFun  implements  an  in-process  game  statistics  engine.  The  engine  measures  and  records 
this  information  on  a  per  Player  basis  for  assessment  review.  The  measures  that  will  be  recorded  in 
PhysicsFun  will  at  a  minimum  record: 

•  Who  the  student  is  -  id,  school,  etc; 

•  What  the  student  did  -  direct  actions,  not  inference  by  game  results; 

•  When  events  occurred  in  game  and  real  time  —  level  completion,  mistakes  made,  self¬ 
corrections,  etc; 

•  Where  (contextually)  in  the  game  the  event  occurred  -  level,  scenario,  specifics  of  play, 
etc. 

The  measure  logs  will  be  used  to  calculate  aggregate  statistics,  such  as  average  time  required 
completing  a  game  level,  number  of  mistakes  made,  and  number  of  self-corrections  made;  to 
provide  an  assessment  of  satisfaction  of  research  objectives. 

5  Interface  Concepts 

The  principal  interface  for  PhysicsFun  will  be  Gesture  based.  There  has  been  a  significant 
amount  of  prior  research  into  the  types  and  effectiveness  of  gesture  based  interfaces(User-Defined 
Gestures  for  Surface  Computing,  April  2009)  (Karam,  November  2006)  (A  taxonomy  of  Gestures  in 
Human  Computer  Interaction,  2005).  We  have  developed  a  basic  taxonomy,  shown  in  Figure  5-1. 
This  taxonomy  will  be  used  to  guide  the  development  of  the  PhysicsFun  interface  and  to  identifying 
potential  alternative  approaches. 
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Manipulative  -  Communicative  - 

Lifecycle  con  trol  or  Stylisti  c  dictionaiy 

man i  pulfiti  on  of  ob  j  nets,  sy mboL i  c  or  semaphore 

.serving  to  communicate 
abstract  concepts. 


Deictic-  Pointing 

Gesticulation- 

to  establish  id  entity 

idiosyncratic,  not 

or  spatial  location  of 

taught,  empty  handed 

an  object 

gestures 

Flow  Form 


Correlation 

Timing 

Pose 

Path 

Independent  Sequential 

Continuous-  Discrete  - 

while  user  acts  alter  user  acts 

Static  -  pose  Dynamic  - 

held  in  one  pose  changes 

shape  shape 

With  Pose  independent 

Figure  5-1  -  Our  simple  taxonomy  of  gesture  types  and  characteristics  provides  aguide  to  further 

investigation  of  interface  approaches. 


We  have  performed  a  preliminary  analysis  of  the  types  and  nature  of  controls  that  will  be 
required  for  the  seedling  game,  mapping  each  game  event  into  the  gesture  taxonomy.  The  result  of 
this  analysis  is  presented  in  the  table  below  which  maps  the  events  from  the  game  map,  Figure  3-2, 
to  the  gesture  type  and  description  of  Figure  5-1. 


Game 

Gesture 

Event 

Type 

Flow 

Form 

Comment 

done 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Complete  and  leave  a  special  mode, 
e.g.  stop  configuring  avatar. 

Fly 

Manipulative  / 
Gesticulation 

Independent 
/  Continuous 

Dynamic 

The  arm  and  leg  motions  used  to 
direct  the  avatars  flight. 

go 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Begin  play  by  entering  the 
appropriate  game  level. 

help 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Go  to  Briefing  Room  for  help. 

jump 

Manipulative  / 
Gesticulation 

Independent 
/  Discrete 

Dynamic 

Jump  from  Aircraft  to  begin  Free 

Falling. 

land 

N/A 

N/A 

N/A 

Automatic  in  game  event. 

leave 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Leave  the  game. 

me 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Begin  avatar  configuration. 
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practice 

Deictic/ 

Communicative 

Independent 
/  Discrete 

Static 

Go  to  Fitting  and  Practice  area. 

pull 

Manipulative  / 
Gesticulation 

Independent 
/  Discrete 

Dynamic 

Open  the  parachute. 

wave- 

off 

Communicative 

Independent 
/  Discrete 

Dynamic 

Abort  the  jump  and  return  to  Aircraft 
or  Briefing  Room 

Two  basic  types  of  gesture  events  have  been  identified.  Game  Control  gestures  which  are 
primarily  communicative,  discrete  and  static  (shaded  blue),  and  Game  Play  gestures  which  are 
primarily  manipulative,  continuous  and  dynamic  (shaded  orange). 

Game  Control  gestures  are  used  to  navigate  and  make  selections  in  game.  These  have 
traditionally  been  performed  using  menu  selections  or  keyboard  entry.  We  are  exploring  several 
options  for  these  types  of  gestures: 

•  Selection  of  icons  by  pointing  to  them  and  hovering  or  using  a  selection  gesture,  such  as 
in  Figure  5-2(a);  and 

•  A  separate  static  gesture,  such  as  a  simplification  of  American  Standard  Sign  language, 
for  example  those  in  Figure  5-2(b). 

The  Game  Play  gestures  represent  a  more  complex  and  diverse  set.  Three  gestures,  jump,  fly 
and  pull,  are  those  which  the  student  uses  to  manipulate 
their  avatar  in  game,  such  as  jumping  out  of  the  aircraft 
to  begin.  These  will  be  based  on  gesticulation,  using  the 
natural  motion  analogous  to  the  desired  avatar  action; 
that  is  the  student  would  jump  up  to  gesture  jump  or 
reach  over  head  and  then  pull  down  to  gesture  pull.  In 
these  two  cases  the  gestures  are  recognized  as  discrete 
events.  The  fly  gesture,  while  perhaps  not  truly  a 
gesticulation  to  the  students,  would  be  using  typical  sky 
diving  positions  to  control  speed  and  direction.  This 
gesture  is  continuous,  monitoring  the  student's  body 
position  during  game  play.  The  fourth  Game  Play  gesture, 
wave-off,  is  used  to  abort  the  game  in  mid-play  and  is  a 
Communicative  rather  than  a  Manipulative  gesture.  An 
important  focus  of  research  is  the  detection  of  these 

discrete  events,  while  monitoring  the  continuous  gestures  (E.G  .fly)  and  ignoring  random 
gesticulation. 


<=>^ 


Point  and  grab 

(a) 

Go  [work] 


Leave (done) 


Figure  5-2  -  Gestures  being  explored 
for  Game  Control;  menu  selection 
(a)  or  standalone  signs(b). 
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5.1  Calibration 

The  calibration  of  real-world  gestures  with  virtual  world  is  required  because  the  students  range 
significantly  in  physical  size.  As  part  of  each  student's  log-in  a  set  of  calibration  data  will  be 
collected  in  order  to  synchronize  real  and  virtual  world  location  references.  This  will  be  done  once 
for  each  new  student  and  optionally  repeated  if  the  need  arises.  Calibration  will  be  accomplished 
by  directing  the  student  player  to  "touch"  a  series  of  object  in  the  virtual  world.  The  actual,  real- 
world  positions  corresponding  to  the  virtual  world  locations  touched  will  be  recorded.  This  will 
establish  a  baseline  calibration. 

During  game  play,  a  "sticky"  object  model  may  be  used  to  provide  continuous  tracking  of 
calibration  during  game  play.  This  process  assumes  that  students  pointing  close  to  an  object  and 
gesturing  to  "select"  it  implies  a  real  world  position  for  the  virtual  object  allowing  fine  tuning  of 
calibration. 

5.2  Voice  Command 

Some  input  devices,  most  notably  the  MS  Kinect,  has  voice  command  capability.  The  potential 
for  combinations  of  deictic  or  manipulative  gestures  with  simple  voices  commands  could 
potentially  provide  a  more  intuitive  interaction.  For  example,  pointing  a  menu  item  and  saying 
"That  one",  rather  than  gesture  alone.  Best  practices  (Microsoft  Coporation,  2012)  indicate  that 
voice  commands  are  most  suitable  for  quiet,  controlled  environments.  Base  on  this 
recommendation,  we  believe  that  voice  controls  are  not  well  suited  to  an  early  elementary  grade 
classroom;  voice  command  will  not  be  investigated  further  in  this  phase  of  the  research. 

5.3  Keyboard  and  Mouse  Command 

While  the  focus  of  the  PhysicsFun  project  is  on  immersive  environments,  we  believe  that  a 
greater  range  of  students  may  be  engaged  by  supporting  traditional  keyboard,  mouse  or  gamepad 
commands  as  well.  Where  possible, 
traditional  equivalent  commands 
will  be  available  for  game  control. 

6  System  Architecture 

PhysicsFun  combines  special 
purpose  input  hardware  and 
computer  gaming  software  to 
provide  an  overall  system 
archetecture  shown  in  Figure  6-1. 

6.1  Hardware  Architecture 

The  core  hardware  used  for 
PhysicsFun  is  a  commercial  off  the 
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shelf  Dell  T1600  Fixed  Workstation  equipped  with  a  quad-core  Xeon  processor  running  Windows  7 
Ultimate.  It  has  been  augmented  to  support  3D  graphics  with  a  ASUS  ENGTX560  graphics  card, 
ASUS  27"  120Hz  LCD  display  and  Nvidia  active  3D  glasses. 

Three  gesture-based  input  devises  are  being  investigated: 

•  AcceleGloves  -  A  pair  of  gloves  with  six  embedded  sensors  per  hand  to  provide  hand  and 
finger  acceleration  measurements, 

•  Microsoft  Kinect  -  A  camera  and  audio  based  system  which  provides  body  skeleton 
tracking  in  three  dimensions  and  directional  sound/voice  recognition, 

•  ASUS  Xtion  PRO  -  A  camera  based  system  which  provides  body  skeleton  tracking. 

Each  device  has  associated  drivers  and  SDKs  which  provide  the  basic  hardware  interface. 

6.2  Software  Architecture 

The  software  architecture  has  two  principal  components.  The  hardware  devices  are  mediated 
through  an  Input  Framework  which 
isolates  the  hardware  details  from  game 
implementation.  The  second  component  is 
the  Gaming  Environment  in  which  the 
actual  game  art  and  behaviors  are 
developed  and  deployed. 


6.2. 1  Input  Framework 

The  input  device  hardware  will  be 
connected  to  the  game  engine  through  an 
Input  Framework.  The  main  purpose  of 
this  framework  is  to  provide,  as  far  as 
possible,  a  device  agnostic  input  stream;  it 
be  based  on  the  OpenNI 5  architecture  as 
shown  in  Figure  6-2.  OpenNI  is  an  open 
source  project  sponsored  by  a  number  of  private  companies  6  interested  in  open  gesture-based 
interfaces. 

OpenNI  is  an  open  source  project  which  seeks  to  provide  exactly  the  hardware  agnostic 
environment  which  is  needed  for  PhysicsFun. 

The  functionality  need  by  the  PhysicsFun  application  will  be  provided  in  middleware  processors 
using  the  OpenNI  APIs  and  data  objects.  These  processors  can  be  organized  at  runtime  into 
processing  strings  to  dynamically  create  different  process  capabilities  and  architectures. 
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Figure  6-2  -  A  generic  architecture  based  on  OpenNI 
allows  the  inclusion  and  evolution  of  hardware  and 
software  components. 


5  http://www.openni.org/ 

6  For  example  PrimeSense  http://www.primesense.com/,  and  ASUS  http://usa.asus.com/ 
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6.2.2  Game  Environment 

The  Game  Environment  is  the  Unity  3D  Integrated  Development  Environment7.  It  is  an 
integrated  development  environment  for  the  development  of  3D  video  games  or  other  real-time 
3D  interactive  applications.  The  Unity  development  environment  will  run  on  both  Microsoft 
Windows  and  Mac  OS  X  platforms;  the  deployment  environments  include  Windows,  Mac,  Xbox 
360,  PlayStation  3,  Wii,  iPad,  iPhone,  Android,  and  Linux.  PhysicsFun  development  will  be  using 
Unity  to  develop  and  deploy  a  standalone  application  for  the  Windows  environment.  Unity 
integrates  the  Input  Framework  using  Microsoft  .NET  dll  libraries  provided  by  the  framework. 
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